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COMPUTER DRAWING SHAPE MANIPULATION WITH ENVELOPE MESHES 

BACKGROUND 

This invention relates computer graphics applications. 

A frequent task in computer graphics applications is to modify the shape of a 

5 graphical object using another graphical object as a guide. This technique is often referred to 
as enveloping. A typical example of enveloping is to use the outline of a shape to shrink and 
elongate letterforms, that is, curves defining the outlines of characters' shapes, to simulate 
the appearance of a banner curling in the wind. The banner shape, which is much simpler 
than the letterforms themselves, can be manipulated using standard point editing controls, 

10 Control points of the letterforms then conform to the general shape of the controlling outline. 
A number of computer graphics applications include an enveloping capability. 
Examples of such computer graphics applications are Freehand® by Macromedia 
Incorporated (San Francisco, California), Corel DRAW® and KPT Vector Effects®, both by 
Corel Corporation (Ottawa, Ontario, Canada). These applications use an "outline-sheet" 

15 metaphor, where the objects to be deformed are placed on the sheet bounded by an envelope. 
The user interacts with the envelope by tugging on its outline. As the envelope is 
manipulated, the details of the objects lying within the envelope move to preserve their 
position relative to the envelope. For example, if a 2-dimensional (2D) u-v coordinate 
system is imposed on the envelope, the u-v positions of objects inside the envelope will be 

20 the same before and after the edit. The envelope can be very simple and have only a few 
control points. An object contained inside the envelope can be edited rather effortlessly, 
even if the object has a multitude of control points. This simple control point editing is the 
primary attraction of enveloping. 

A number of three-dimensional (3D) modeling applications include lattice 

25 deformations that are the 3D equivalents of envelopes. Examples of such applications are 

Softimage® by Softimage Corporation (Montreal, Quebec, Canada), and Alias/Wavefront 

Maya® by Alias/Wavefront Incorporated, a division of Silicon Graphics (Mountain View, 

California). In lattice deformations, an m x n lattice of envelopes is arranged in three- 

l 



dimensional space. Typically the lattice is a closed lattice, forming a tube-like structure 
around a 3D object. Similar to the envelopes in a 2D drawing application, the user can 
manipulate the lattice, which is a fairly simple 3D object, and thereby change a more 
complex object lying inside the lattice. 
5 A number of computer image processing applications include a variant of envelopes 

to perform image deformations. One example of such an application is described by Ron 
Brinkmann in "The Art and Science of Digital Compositing" (pages 58-63, Morgan 
Kauftnann publishers, 1999). In these systems, an m x n grid of envelopes is superimposed 
over an image, thus defining the default mapping of the image pixels into the u-v coordinate 
1 o system of the grid. A duplicate copy of the image and grid pair (the destination pair) is 

typically displayed beside the original pair (the source pair). These systems allow either the 
source or destination grids to be manipulated. When either grid is manipulated, the 
^ destination image is regenerated based on preserving the u-v positions of the image pixels 

SB relative to the destination grid. This is analogous to the regeneration of the contained object 

S 15 in the drawing applications discussed above. 

y f Even though the drawing applications described above provide a flexible working 

Cp environment for the users, they do exhibit some shortcomings, which are addressed by the 

r% present invention. 

H SUMMARY 

p 20 In general, in one aspect, this invention provides methods and apparatus, including 

computer program products, implementing and using techniques for performing 
envelope-based computer graphics operations on vector objects in an image. An envelope in 
an image generally contains an original drawing object. The envelope is a manipulable 
graphic object defining a coordinate remapping that is applied to generate a resulting drawing 

25 object for any original drawing object in the envelope. In one embodiment, the envelope has 
an interior control point that is an anchor point. User manipulation of the interior control 
point redefines the coordinate remapping. In another embodiment, a precision variable 
determines how closely an object in the envelope will follow the envelope when the 
corresponding resulting object is generated. Additional control points can be added to the 

30 original contained object if necessary to achieve the required precision before applying the 
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coordinate remapping. In another embodiment, CI (first derivative) or C2 (second 
derivative) continuity at curve anchor points is preserved over envelope manipulations. 

The invention can be implemented to realize one or more of the following 
advantages. A higher degree of control over shape manipulation capabilities of the envelopes 
can be achieved. Interior control, local refinement of meshes, source envelopes, variable 
precision of distortion, curve continuity preservation, isolation of editing focus and preset 
visual cues provide a higher level of artistic expression and usability to computer drawing 
application enveloping compared to conventional computer drawing applications. Interior 
control does not have to be rectilinear, which allows flexibility and simplicity in control of 
fine detail A user can isolate the source art or envelope for editing, which is advantageous 
when they overlap. The coordinate remappings are managed in a way that prevents loss of 
curve tangent and curvature continuity. 

The details of one or more embodiments of the invention are set forth in the 
accompanying drawings and the description below. Other features, objects, and advantages 
of the invention will be apparent from the description and drawings, and from the claims. 

DESCRIPTION OF DRAWINGS 
FIG. 1 is a block diagram showing a general process for modifying objects using an 
envelope. 

FIG. 2 is a schematic illustration of an envelope mesh with interior control points in 
accordance with the invention. 

FIG. 3 is a schematic illustration of an envelope mesh with additional control points 
in accordance with the invention. 

FIG. 4 is a schematic illustration of an envelope having different precision values set 
in accordance with the invention. 

FIG. 5 is a schematic illustration of curve continuity preservation in accordance with 
the invention. 

FIG. 6 is a schematic illustration of an interface showing preset visual cues and 
editing focus in accordance with the invention. 

Like reference symbols in the various drawings indicate like elements. 



DETAILED DESCRIPTION 
The invention will now be described in greater detail by way of example. First, a 
general description of the application architecture will be given. This general description 
will be followed by individual sections for each of the following features: interior control 
points, local refinement, source envelopes, editing isolation, precision, curve continuity 
preservation, and preset visual cues. In the following description, the shape to be distorted 
will be referred to as "art." The guiding shape to which the art will be distorted will be 
referred to as an "envelope". The term envelope encompasses both an outline-only envelope 
as well as an envelope, such as a mesh, that has interior control. An envelope may be 
referred to as a "mesh" or an "envelope mesh" to emphasize that in a particular situation both 
an outline and interior points influence the art's shape. A combination of art and envelope 
mesh may be referred to as an "envelope group." 

In order to explain the term mesh, it will be useful to start with the definition of a 
grid. A grid is an array of m x n control points or lattice points. Each lattice point in the grid 
has a position and four tangents, one in each of the positive and negative u and v directions, 
respectively. One exception is lattice points on the edge of the grid, which have fewer 
tangents. The position of the tangent vectors define controlling curves connecting each pair 
of adjacent control points in the grid. A mesh is a grid that may have extra control points 
between the existing lattice points. These extra control points are defined by a position, also 
referred to as anchor point, and two tangents. The tangents are either in the positive and 
negative u direction or in the positive and negative v direction, depending on whether the 
curve on which the extra control point is located connects points that have a common u 
coordinate or a common v coordinate. 

The present implementation is not a stand alone drawing application, but rather an 
addition to an existing drawing application, such as Adobe® Illustrator®, which features an 
extensible and open plug-in architecture. The existing application, or host application, 
includes standard drawing application features such as hierarchical object list management, 
shape creation utilities, and object control point editors. Furthermore, the host application 
includes a feature for creating color transition meshes. 

Conventional enveloping applications typically embed an envelope and the object or 
objects contained inside the envelope in a hierarchal object list so that the objects can inherit 



the behavior of standard drawing application objects. Examples of such behavior include 
precedence in the object list, colorization and printing. The application operating in 
accordance with the invention is no different in this respect. As shown in FIG. 1, a 
placeholder, or group, is created in the object list (105). The group is populated with the 
original contained objects (also referred to as the source art or source objects), the envelopes 
(source and destination), and versions of the contained objects which have undergone the u-v 
coordinate remapping (result objects) (110). A messaging protocol is then established (115), 
which ensures that the result objects are regenerated whenever the envelopes or contained 
objects are modified. The procedure checks if an envelope or an object contained inside the 
envelope has been modified (120). If not, the page with the objects are displayed (145). If 
an envelope or an object has been modified, the envelope or object notifies the group when it 
has been edited (125) and the group in turn accesses the source objects (130), of which there 
can be many, and creates result objects by successively applying the coordinate remapping to 
them (135). Finally, the result objects are displayed (140). 

In one implementation, the source envelope is implicit, so that it is not explicitly 
placed by the user or displayed for modification by the user. In this case, the source envelope 
is defined by the bounding box of the source art, in whatever way that is defined by the 
application. For example, when the user selects one or more objects and applies an envelope 
to the objects, the bounding box of the objects defines the source envelope or mesh and the 
user's selection and the user's subsequent editing, if any, defines the destination envelope. If 
the source envelope is implicit to the user, it can be represented implicitly in the envelope 
group as well, as it can always be derived from the source art. 

Interior Control Points 

There are two distinct methods of introducing interior control points into envelopes, 
which will be discussed in this and the next section. In the first method, meridians of 
constant u or v are introduced into the envelope, as shown in FIG. 2. The number of control 
points inserted in u or v corresponds to the dimension of the other variable. Thus, if a u 
meridian (200) is added to a 3x4 grid to create a 4x4 grid, four new control points (201, 202, 
203, 204) will be added. This creates a grid-like matrix of envelope control points similar to 
the control points found in image processing applications. 



Local Refinement 

The second method of introducing interior control points avoids introducing an entire 
row or column of control points. Instead, single additional control points are introduced. As 
can be seen in FIG. 3, these single additional control points (301, 302) are localized to 
different portions of the envelope between the existing control points (303, 304). In so 
doing, details in the source object can be isolated for more precise user control of the u-v 
coordinate remapping. In the application in accordance with the invention, the user chooses 
a tool that adds control points and indicates where on the mesh the insertion should be made. 
The indicated location becomes a control point and can then be manipulated identically to 
any other envelope control. 

Source Envelopes 

In the image processing applications described in the background section, it was 
noted that a source and destination mesh provide a higher degree of control of deformations 
than a destination mesh alone. It was also pointed out that in the image processing 
application the source and destination envelopes are typically displayed side by side. This 
side-by-side editing is not feasible in a drawing application, because the object being 
enveloped typically resides within the context of an entire page description. Therefore, in the 
application in accordance with the invention, the source envelope resides in the group that 
handles the messaging between envelopes and contained objects, thus having a dedicated 
position within the page's hierarchical object list. By being in this position, the effects of 
edits to the source envelope can be seen in relationship to the other objects which will appear 
in the final printed page. 

In the drawing application in accordance with the invention, every page position lying 
inside the source mesh can be mapped to the u-v coordinate system of the source mesh. The 
u-v positions of the source object's control points are defined by this mapping of page 
positions. In the destination mesh, the u-v coordinate values of the control points are held 
constant, but since the destination mesh is different than the source mesh, the location of the 
control points on the page will be different. The destination object is generated 
programmatically based on the source object and the source and destination meshes, 
respectively, so a user never interacts directly with the destination object. 



Editing Isolation 

The control points of contained objects in standard enveloping applications typically 
reside within the extent of the envelope. Furthermore, in the outline-sheet model, only the 
outlines of the envelope influence the regeneration of the contained object. As a 
5 consequence, the control points of the contained object and the control points of the envelope 
seldom overlap, and the user's intention with respect to manipulating a particular control 
point is clear. 

When interior controls and source envelopes are introduced, the user's intention is not 
always clear. The application in accordance with the invention provides two mechanisms for 

10 establishing the focus of interactive editing. Typical drawing application point editing tools 
honor the front to back ordering of the page, that is, objects which appear on top of other 
objects will be treated preferentially when the application detects which control point a user 
clicks at. The application in accordance with the invention has a variable front to back 
ordering of the source objects and envelopes, under user control, specific to that purpose. 

15 Conventional, built-in drawing application commands such as move forward, or move 

backward in the hierarchical object list are not well suited to this task, as the envelopes are 
generally rendered as outlines and it is unclear from a screen rendering if the source mesh, 
destination mesh, or the source shape is closest to the front. The source shape refers to either 
a single source object or an arbitrarily large collection of source objects. The user control 

20 interface in accordance with the invention simply displays a list of the three choices (source 
mesh, destination mesh, or source shape) and asks which choice should have precedence. In 
FIG. 6, a version of a user interface (600) is shown in which the option for a user to interact 
with the source mesh is disabled, so the user can only choose between letting the source 
shape or the destination mesh be at front by selecting an option from a popup menu (605). 

25 Therefore the destination mesh is referred to simply as the "mesh" in the FIG. 6. 

A second mechanism of indicating preference is to hide or lock certain elements in 
the envelope group. Elements that are hidden or locked cannot respond when a user clicks at 
a control point. A combination of these two techniques yields satisfactory results, that is, 
establishes a rule of hiding or locking the objects that are not indicated as the focus of 

30 editing. This can be performed automatically or be left as an option to be selected by a user. 
In FIG. 6 the locking option is selected by clicking on the padlock symbol (610). 
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Curve Continuity Preservation 

Tangent and curvature continuity are distinctive properties of the curves used in 
computer drawing applications. The design of character glyphs is particularly focused on 
establishing forms that are visually appealing. Curve definitions in drawing applications are 
generally in the form of cubic polynomial functions of four adjacent control points. Typical 
examples include Bezier, Hermite and B splines, with Bezier being the most prevalent. The 
mathematical formulation of Bezier curves does not guarantee specific tangent and curvature 
continuity in the way other formulations, such as B splines do. For example, with Bezier 
splines, achieving tangent continuity requires three adjacent control points to be co-linear. 
With the introduction of interior controls in the envelope, it is not unusual for this 
relationship of control points to be broken. 

The control points in computer drawing applications are typically differentiated as 
anchor points and tangent handles. Each control point having a single anchor point (p) (501), 
a preceding tangent (502) bounded by a tangent handle (in) (503), and a following tangent 
(504) bounded by a tangent handle (out) (505), as can be seen in FIG. 5. With this user 
interface paradigm, a four point Bezier spline is defined by two neighboring anchor points, 
the out tangent of the first anchor point, and the in tangent of second anchor point. 

In one implementation of the invention, tangent or curvature continuity for a Bezier 
spline is preserved by: 

(1) storing the continuity state of each anchor point and the ratio of the anchor point 
relative to the tangent handles; 

(2) remapping the tangent handles in accordance with the distortion field; and 

(3) relocating the anchor point. 

If the curve tangent or curvature is not continuous at the control point, the anchor 
point simply moves according to the distortion field, independent of the remapped tangent 
handles. If the curve tangent or curvature is continuous at the point, the anchor point is 
moved so that the ratio of the anchor point's position relative to the remapped tangent 
handles is identical to the ratio prior to remapping. This method insures the preservation of 
the control point's original continuity. 



Precision 

The distortion of the contained object is accomplished by remapping the Cartesian 
positions of curve controls such that u-v positions are preserved in distorted and undistorted 
spaces. With envelope meshes, subtle distortion effects can be created, such as rippling 
water To provide the artistic control necessary to get a variety of effects from a single 
envelope, it is necessary to introduce a precision variable that determines how closely the 
contained object will follow the envelope. This behavior is accomplished by introducing 
additional control points on the original curves prior to the coordinate remapping, with a 
variable frequency of insertion determined by the user. The higher value of the precision 
variable, the better the control points follow the curve. An example can be seen in FIG. 4, 
where the precision variable has the values 1 (401), 3 (402) and 10 (403), respectively. In 
this example, the precision variable is used to represent the number of additional anchor 
points inserted between the original anchor points of the source art. A second benefit of 
having the variable insertion under user control is that the performance of distortion of large 
amounts of data can be reasonably managed. 

Preset Visual Cues 

Drawing applications often include collections of pre-made envelopes. 
Conventionally, these presets are presented pictorially in a window where a user may create 
an envelope simply by selecting a swatch that has been rendered with a representation of the 
envelope. The rendering of the envelopes generally consists of line drawings of the 
controlling outlines. With envelope meshes, this type of rendering is insufficient. For 
example, two 3x3 envelopes that differ only by the position of the central control point will 
appear identical. A preferable approach is to render an envelope group into the swatch 
whose contained object possesses interior details that have a familiar structure. An example 
of an object with such interior details is a collection of squares of two or more colors 
arranged in a checkerboard pattern, as can be seen in FIG. 6. 

The invention can be implemented in digital electronic circuitry, or in computer 
hardware, firmware, software, or in combinations of them. Apparatus of the invention can be 
implemented in a computer program product tangibly embodied in a machine-readable 



storage device for execution by a programmable processor; and method steps of the invention 
can be performed by a programmable processor executing a program of instructions to 
perform functions of the invention by operating on input data and generating output. The 
invention can be implemented advantageously in one or more computer programs that are 

5 executable on a programmable system including at least one programmable processor 

coupled to receive data and instructions from, and to transmit data and instructions to, a data 
storage system, at least one input device, and at least one output device. Each computer 
program can be implemented in a high-level procedural or object-oriented programming 
language, or in assembly or machine language if desired; and in any case, the language can 

10 be a compiled or interpreted language. Suitable processors include, by way of example, both 
general and special purpose microprocessors. Generally, a processor will receive instructions 
and data from a read-only memory and/or a random access memory. Generally, a computer 
will include one or more mass storage devices for storing data files; such devices include 
magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and 

15 optical disks. Storage devices suitable for tangibly embodying computer program 

instructions and data include all forms of non-volatile memory, including by way of example 
semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; 
magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and 
CD-ROM disks. Any of the foregoing can be supplemented by, or incorporated in, ASICs 

20 (application-specific integrated circuits). 

To provide for interaction with a user, the invention can be implemented on a 
computer system having a display device such as a monitor or LCD screen for displaying 
information to the user and a keyboard and a pointing device such as a mouse or a trackball 
by which the user can provide input to the computer system. The computer system can be 

25 programmed to provide a graphical user interface through which computer programs interact 
with users. 

The invention has been described in terms of particular embodiments. Other 
embodiments are within the scope of the following claims. For example, the steps of the 
invention can be performed in a different order and still achieve desirable results. 
30 What is claimed is: 
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